Topic words analysis based on LDA model

摘要

Social network analysis (SNA), which is a research field describing andmodeling the social connection of a certain group of people, is popular amongnetwork services. Our topic words analysis project is a SNA method to visualizethe topic words among emails from Obama.com to accounts registered in Columbus,Ohio. Based on Latent Dirichlet Allocation (LDA) model, a popular topic modelof SNA, our project characterizes the preference of senders for target group ofreceptors. Gibbs sampling is used to estimate topic and word distribution. Ourtraining and testing data are emails from the carbon-free serverDatagreening.com. We use parallel computing tool BashReduce for word processingand generate related words under each latent topic to discovers typicalinformation of political news sending specially to local Columbus receptors.Running on two instances using paralleling tool BashReduce, our projectcontributes almost 30% speedup processing the raw contents, comparing withprocessing contents on one instance locally. Also, the experimental resultshows that the LDA model applied in our project provides precision rate 53.96%higher than TF-IDF model finding target words, on the condition thatappropriate size of topic words list is selected.